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24,000  to  14,000  employees.  The  Contract  Administration  Offices  that  make  up  DCMC 
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efficiency  of  these  Contract  Administration  Offices  in  the  wake  of  this  massive 
reorganization.  Statistical  analysis  determined  the  most  crucial  inputs  and  outputs  for  use 
in  an  efficiency  model.  Data  Envelopment  Analysis  (DEA)  methodology  identified 
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EXECUTIVE  SUMMARY 


The  Defense  Contract  Management  Command  (DCMC)  has  undergone  a 
significant  reorganization  in  recent  years.  The  number  of  employees  within  the 
command  has  reduced  from  24,000  in  1990  to  14,000  today.  It  is  unknown,  however, 
whether  this  reorganization  resulted  in  an  efficient  organization.  This  study  determines 
the  efficiency  of  the  65  individual  Contract  Administration  Offices  located  throughout 
the  United  States  that  are  assigned  to  DCMC. 

Part  of  the  Defense  Logistics  Agency,  DCMC  provides  contract  management 
services  for  most  contracts  awarded  within  the  Department  of  Defense;  over  350,000 
contracts  managed  at  any  given  time.  In  order  to  manage  the  implementation  of  these 
contracts.  Contract  Administration  Offices  (CAO)  provide  continuous  contract 
management  services  for  government  program  managers  overseeing  individual 
manufacturers  under  contract.  CAOs  provide  support  for  fact-finding  and  negotiations 
between  contractors  and  contracting  officers,  provide  safety  and  environmental 
assurance,  and  evaluate  contractor  processes  and  progress,  among  other  duties.  Two 
types  of  CAOs  operate  within  DCMC:  Plant  Offices  and  Area  Offices.  Area  Offices 
manage  contracts  within  a  geographic  area.  Plant  Offices  are  resident  within  contractors’ 
plants.  Because  of  the  inherent  differences  in  the  two  CAO  types,  management  has 
determined  that  each  office  type  be  modeled  separately. 

Efficiency  is  a  ratio  of  outputs  to  inputs.  However,  difficulties  arise  in  this  simple 
concept  when  attempting  to  compare  efficiency  between  units  without  easily  identifiable 
or  countable  inputs  and  outputs  or  when  units  have  multiple  inputs  or  outputs.  The 


xm 


methodology  known  as  Data  Envelopment  Analysis  or  DEA,  uses  mathematical 
programming  techniques  to  combine  multiple  output  and  input  measures  to  produce  a 
single  efficiency  score  for  each  organization. 

The  selection  of  measures  for  use  in  the  models  is  of  utmost  importance  when 
implementing  DEA  models.  Evidence  of  a  causal  relationship  between  inputs  and 
outputs  must  be  shown  though  analysis  of  their  associated  correlation  coefficients. 
DCMC  provided  one  candidate  input  measure  and  eight  candidate  output  measures  of 
which  only  six  output  measures  are  selected  for  use  in  the  models.  Of  those  six,  three  fail 
to  show  strong  evidence  of  this  causal  relationship.  However,  those  three  measures,  the 
only  measures  of  quality  available,  are  deemed  essential  to  the  overall  formulation  of  the 
DEA  models,  even  though  their  low  correlation  coefficient  with  the  input  measure 
weakens  the  final  efficiency  results. 

Fifteen  of  the  65  offices  are  shown  to  be  efficient.  Inefficient  offices  can 
determine  improvement  opportunities  by  examining  the  measures  of  efficient  offices.  To 
improve,  an  office  must  either  lower  its  input,  increase  output,  or  both.  The  resulting 
scores  show  that  some  offices  are  achieving  a  greater  number  of  outputs  per  input 
compared  to  other  offices.  Efficient  offices  can  be  used  as  a  benchmark  for  other  offices 
to  emulate. 

The  DEA  models  in  this  study  provide  a  valuable  management  tool  to  assist 
DCMC  in  determining  personnel  resource  requirements.  The  great  added  value  with  the 
DEA  models  is  the  new  visibility  over  each  office's  strengths  and  weaknesses.  The  rapid 


identification  of  these  weaknesses  allows  management  to  focus  on  problems  and  solve 
them  quickly  without  a  great  time  investment  searching  for  possible  causal  factors. 
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I.  INTRODUCTION 


...We  tend  to  meet  any  new  situation  by  reorganizing:  a  wonderful  method 
...  for  creating  the  illusion  of  progress,  while  producing  ...inefficiency." 

--  Petronius  Arbiter,  210  BC 

A.  OVERVIEW 

By  any  measure,  the  Department  of  Defense  has  undergone  dramatic 
reorganization  in  recent  years.  Less  bureaucracy  and  smaller  organizations  are  always 
assumed  to  be  better  and  in  the  best  interest  of  the  taxpayers.  This  also  assumes  that 
governmental  organizations  are  producing  the  same  services  as  in  the  past  but  with  less 
"overhead"  or  cost,  although  relatively  few  studies  can  support  this  claim.  In  fact,  little 
evidence  exists  to  support  the  notion  that  downsizing  and  reorganization  within  the 
defense  department  has  produced  more  efficient  organizations. 

The  Defense  Contract  Management  Command  (DCMC),  part  of  the  Defense 
Logistics  Agency,  is  charged  with  managing  a  staggering  number  of  defense  contracts. 
Along  with  nearly  all  defense  related  organizations,  DCMC  has  seen  its  personnel  rolls 
drop  steadily  since  1990.  This  reorganization  should  have  resulted  in  a  lean,  efficient 
command.  The  efficiency  of  the  65  Contract  Administration  Offices  based  in  the  United 
States  that  make  up  DCMC  is  the  subject  of  this  thesis. 

In  its  most  basic  form,  efficiency  is  a  ratio  of  products  (or  outputs)  to  inputs. 
Regardless  of  the  type  of  output  and  input,  resulting  efficiency  is  simply  computed  as 
outputs  per  input  unit.  For  example:  "Automobiles  per  dollar"  is  an  effective  means  to 
compare  efficiency  between  automobile  factories  producing  similar  products. 
Difficulties  arise  quickly  when  attempting  to  compare  efficiency  between  units  without 
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easily  identifiable  or  countable  inputs  and  outputs.  Products  from  "white  collar"  workers 
such  as  managers  and  clerical  staff  are  far  more  difficult  to  quantify.  In  private 
enterprise,  profit  is  always  available  for  use  as  the  measured  output.  Because  of  the  lack 
of  a  profit  statement  in  the  public  sector,  these  organizations  are  often  characterized  by 
multiple  outputs  and  occasionally  multiple  inputs  (Ganley,  1992,  p.2).  The  very  nature  of 
public  sector  efficiency  analysis  lends  itself  directly  to  the  methodology  known  as  Data 
Envelopment  Analysis  or  DEA,  which  combines  multiple  output  and  input  measures  to 
produce  a  single  efficiency  score  for  each  organization.  DEA  is  the  basis  for  the 
efficiency  analysis  of  the  65  DCMC  Contract  Administration  Offices. 

B.  BACKGROUND 

Defense  Contract  Management  Command  (DCMC)  provides  contract 
management  services  for  diverse  products  and  services  for  most  contracts  awarded  within 
the  Department  of  Defense.  DCMC  manages  over  350,000  contracts  with  23,000 
contractors  worldwide  totaling  over  $900  billion  in  unliquidated  contractual  obligations 
and  employs  over  14,000  employees  (Defense  Contract  Management  Command,  1998). 

In  order  to  manage  the  implementation  of  these  contracts  after  their  award,  DCMC, 
headquartered  in  Fort  Belvoir,  Virginia,  is  organized  into  three  major  districts:  Eastern 
District,  Western  District,  and  International  District.  Each  District  is  responsible  for  the 
control  and  supervision  of  several  Contract  Administration  Offices  (CAO)  which  provide 
continuous  contract  management  services  for  government  program  managers  overseeing 
individual  manufacturers  under  contract  to  the  Department  of  Defense  (Defense  Contract 
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Management  Command,  1998).  In  this  analysis,  only  Eastern  and  Western  Districts  are 
analyzed. 

A  total  of  65  CAOs  actively  administer  contracts  for  the  districts  in  DCMC. 
Contract  Administration  Offices  provide  support  for  fact-finding  and  negotiations 
between  contractors  and  contracting  officers,  provide  safety  and  environmental 
assurance,  evaluate  contractor  processes  and  controls,  evaluate  contractor  corrective 
actions,  provide  control  of  government  property,  and  provide  an  independent  evaluation 
of  contractor  progress  (Defense  Logistics  Agency,  1998). 

In  general,  both  the  number  of  active  offices  in  DCMC  and  the  overall  staffing 
level  at  each  Contract  Administration  Office  have  reduced  since  1990  due  to  Defense 
Department  downsizing.  Because  of  this,  proper  resourcing  of  Contract  administration 
Offices  has  come  into  question.  It  is  not  known  if  individual  CAOs  are  over  or 
understaffed  with  respect  to  their  assigned  workload  (Boyce,  1998). 

Two  types  of  CAOs  operate  within  DCMC.  Defense  Contract  Management  Area 
Offices  (DCMAO)  manage  contracts  within  a  geographic  area.  Defense  Plant 
Representative  Offices  (DPRO)  are  resident  within  contractors’  plants  and  are 
responsible  for  administration  services  with  that  particular  contractor  only.  Contract 
management  includes  daily,  on-site  surveillance  of  contractor  systems  and  program 
specific  concerns  that  cannot  be  viewed  or  monitored  by  off-sight  agencies.  The  CAO 
organization  allows  DCMC  personnel  to  reside  close  to  or  in  contractors’  facilities  and  to 
tailor  their  services  to  specific  customers  (DLA,  1998). 

Of  the  65  CAOs,  nearly  half  are  DPROs  and  half  DCMAOs  (34  and  31 
respectively).  The  greatest  difference  between  area  offices  and  plant  offices  is  the  size 
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and  number  of  contracts  managed.  Area  offices  tend  to  manage  many  (sometimes 
thousands)  of  contracts  of  limited  size  and  scope.  Plant  offices  manage  fewer  contracts 
with  a  specific  contractor  but  each  contract  generally  has  a  larger  value  and  associated 
complexity.  For  this  reason,  measures  of  production  output  for  these  two  types  of  offices 
are  quite  different.  Area  offices  tend  to  measure  output  by  quantity  of  contracts  managed 
while  plant  offices  tend  to  use  the  size  or  dollar  value  of  the  contract  managed. 

Because  of  the  inherent  differences  in  the  two  CAO  types,  (different  management 
techniques,  volume,  type,  and  complexity  of  work),  DCMC  management  has  determined 
that  each  office  type  should  be  modeled  separately.  The  two  models  are  referred 
throughout  this  text  as  the  Plant  Office  Model  for  DPROs  and  Area  Office  Model  for 
DCMAOs. 

As  will  be  shown  in  following  chapters,  the  methodology  known  as  Data 
Envelopment  Analysis  (DEA),  a  mathematical  programming  technique,  provides  a 
methodology  to  determine  efficiency  among  public  sector  organizations  with  multiple 
input  and  output  measures  (Norman,  1991,  pp.  5-8).  When  implementing  DEA  models,  a 
great  deal  of  care  must  be  taken  when  selecting  measures  for  inclusion  in  the  model. 

Data  analysis  techniques  of  the  potential  input  and  output  measures  guide  the  modeler  to 
select  not  only  the  proper  measures  for  the  model,  but  also  the  appropriate  DEA  model 
formulation. 

Once  the  measures  and  specific  DEA  model  are  selected,  an  efficiency  score  is 
computed  for  each  office.  These  scores  not  only  identify  inefficient  and  efficient  offices, 
they  also  suggest  potential  improvement  strategies  for  inefficient  offices. 
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In  Chapter  II,  a  brief  discussion  of  Data  Envelopment  Analysis  and  its  relevance 
for  this  study  is  investigated.  Chapter  HI  determines  which  potential  input  and  output 
measures  the  proper  attributes  for  inclusion  in  a  DEA  model.  Chapter  IV  shows  the  final 
formulation  of  the  DEA  model  and  discusses  the  results  as  well  as  insights  into  the 
sensitivity  of  the  model  and  potential  improvements  for  inefficient  offices. 
Recommendations  and  conclusions  are  provided  in  Chapter  V. 
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II.  METHODOLOGY:  DATA  ENVELOPMENT  ANALYSIS 


A.  EVALUATING  CONTRACT  ADMINISTRATION  OFFICES 

The  combined  effect  of  Defense  Department  downsizing  and  subsequent 
reduction  in  defense  related  acquisition  of  goods  and  services  has  been  reduction  in  both 
the  number  and  size  of  contracts  managed  by  DCMC.  Some  CAOs  have  consolidated  or 
closed,  sometimes  increasing  the  workload  at  a  CAO,  but  in  general,  workload  has 
decreased  (Boyce,  1995,  p.  11). 

Proper  staffing  of  individual  CAOs  at  DCMC  has  long  been  a  subject  of  debate. 
No  effective  means  of  scientifically  determining  the  proper  staffing  level  existed  until 
very  recently.  In  1995,  the  Defense  Logistics  Agency  Office  of  Operations  Research  and 
Resource  Analysis  fit  a  linear  regression  model  of  staffing  as  a  function  of  workload 
(Boyce,  1995).  Proper  resourcing  was  then  taken  to  be  the  value  of  the  predicted  staffing 
from  the  regression  line  for  a  particular  workload.  Two  separate  regression  models,  one 
for  plant  offices  and  one  for  area  offices  were  used  to  determine  staffing.  A  feature  of 
these  regression  models  is  that  a  few  offices  with  great  influence  may  induce  an  improper 
target  staffing  level  (either  too  high  or  too  low).  Using  regression  also  assumes  the 
notion  that  an  "average"  office  exists  without  taking  into  account  the  unique  missions  of 
different  offices.  Most  importantly,  the  regression  methodology  does  not  clearly  identify 
offices  that  are  particularly  efficient  and  establish  targets  for  the  remaining  offices  to 
reach.  In  fact,  the  regression  model  could  lead  decision  makers  to  add  personnel 
resources  to  an  efficient  office  because  its  outputs  per  unit  input  do  not  conform  to  the 
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average.  A  better  response  is  to  identify  the  efficient  CAOs  as  the  benchmark  for  other 
offices  to  emulate. 

Determining  relative  efficiency  between  these  offices  a  better  method  for 
comparing  their  performance.  If  every  CAO  produced  exactly  the  same  products,  simple 
ratios  of  inputs  and  outputs  could  be  used  to  compute  relative  efficiencies  easily.  The 
offices  can  be  ranked  by  their  efficiency  of  outputs  /  unit  input.  This  metric  gives  great 
insight  into  those  offices  with  the  best  business  practices.  If  one  “best  practice”  office 
can  gain  two  outputs  per  unit  input,  surely  all  offices  should  strive  to  attain  that  goal. 

However,  a  fundamental  problem  arises  when  more  than  one  input  or  output  is 
necessary  for  relative  efficiency  comparisons.  It  is  possible  that  an  office  may  produce  a 
lot  of  one  output  but  relatively  little  of  another.  A  composite  efficiency  score  can  be 
computed  by  taking  the  mean  efficiency  score.  Alternatively,  management  could  assign  a 
weight  or  relative  importance  to  each  output  or  input,  thereby  developing  a  weighted 
average  efficiency  for  each  office.  Unfortunately,  both  of  these  methods  impose  certain 
assumptions  on  the  underlying  data.  In  the  first  case,  each  output  and  input  is  assumed  to 
be  equally  vital  to  each  office’s  overall  efficiency.  In  the  case  of  management 
determined  weights,  the  assumption  is  that  those  heuristically  developed  weights  are 
valid  and  that  they  apply  to  every  office.  The  “one  size  fits  all”  approach  is  not  always 
appropriate.  In  particular,  for  organizations  that  may  have  different  contexts  or 
environments  in  which  to  work,  there  may  be  great  variability  among  output  and  input 
measures. 

What  truly  is  required,  in  order  to  find  a  fair  measure  of  an  office’s  efficiency,  is 
an  efficiency  score  that  is  optimal  for  that  office:  an  efficiency  score  which  has  the  best 
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weight  on  each  output  and  input  so  that  an  individual  office  achieves  the  highest  possible 
score.  This  provides  each  office  with  an  evaluation  of  its  inherit  efficiency  based  on  its 
strengths.  This  is  exactly  how  Data  Envelopment  Analysis  computes  relative  efficiency 
scores  with  multiple  inputs  and  outputs  and  why  it  offers  an  excellent  methodology  to 
determine  the  efficiency  of  Contract  Administration  Offices. 

B.  DATA  ENVELOPMENT  ANALYSIS 

1.  Overview  of  the  Methodology 

Data  Envelopment  Analysis  (DEA)  is  a  mathematical  modeling  methodology  that 
determines  a  Decision-Making  Unit’s  (DMU)  efficiency  using  non-linear  programming 
techniques.  (Because  each  non-linear  program  is  a  fractional  program,  it  can  be  easily 
translated  into  an  equivalent  linear  program.)  Decision  Making  Units  are  production 
nodes  that  transform  the  inputs  into  outputs.  For  the  purpose  of  this  research,  CAOs  are 
the  DMUs. 

Originally  developed  by  Chames,  Cooper,  and  Rhodes  in  1978  (Chames,  1994, 
pp.  23-25),  DEA  models  calculate  an  efficient  frontier  by  comparing  each  DMU’s  inputs 
and  outputs.  The  efficient  frontier  is  composed  of  DMUs  that  have  attained  the  maximal 
efficiency  score  of  one.  All  other  offices,  those  that  are  inefficient,  have  an  efficiency 
score  less  than  one  and  are  not  on  the  frontier.  The  DEA  model  determines  a  relative 
efficiency  score  for  each  DMU  which  represents  how  far  the  DMU  is  from  an  efficient 
frontier  (Norman,  1991,  p.  43). 

In  order  to  calculate  the  efficiency  of  a  particular  DMU,  mathematical 
programming  techniques  are  used  to  determine  “virtual  multipliers”  or  “weights”  for  the 
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relative  value  of  the  various  outputs  and  inputs  that  maximize  a  specific  DMU’s 
efficiency  score.  The  weights  are  thus  the  primal  variables  in  the  mathematical  program. 
A  particular  DMU  may  utilize  any  combination  of  inputs  and  outputs  in  order  to 
maximize  its  own  efficiency  score  subject  to  the  constraint  that  all  other  DMUs’ 
efficiency  scores  using  that  particular  DMU’s  weights  are  less  than  or  equal  to  one.  A 
separate  linear  programming  formulation  is  used  to  calculate  the  efficiency  score  for  each 
DMU.  It  is  important  to  note  that  DEA  models  produce  only  relative  efficiency  scores  in 
comparison  to  all  other  DMUs.  Unlike  regression  techniques  that  estimate  an  average 
production  function,  DEA  techniques  identify  two  subsets  of  DMUs:  the  efficient  and 
inefficient. 

2.  Model  Formulations 

The  two  most  prevalent  DEA  model  formulations  are  the  model  developed  by 
Chames,  Cooper,  and  Rhodes  (1978),  and  the  model  developed  by  Banker,  Chames,  and 
Cooper  (1984),  called  the  CCR  and  BCC  models  respectively.  The  most  simple,  the 
CCR,  is  formulated  in  Figure  1. 
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Indices: 

i 

Output  Types  (1,2, ...I) 

j 

Input  Types  (1,2,...J) 

k 

DMUs  (1,2,...K) 

Data: 

Xkj 

Value  of  input  j  for  DMU  k 

yki 

Value  of  output  i  for  DMU  k 

s 

A  small  constant. 

Variables: 

vi 

Weight  on  input  j  (unitless) 

Wi 

Weight  on  output  i  (unitless) 

Formulation: 

For  each  DMU  o,  (o  =  1,2,. ..K) 

Y  y0iWi 

Maximize  -=L= - 

la  XojVj 
i 

Subject  to  constraint: 

X  yhWi 

‘  <1,  fork  =  l,2,...K 

2,  w 

j 

Wj>£,  for i  =  1,2, ...I 
vj>£,  for  j  =  1,2,... J 

Figure  1.  CCR  DEA  Model  Formulation  (Norman,  1991,  p.  235) 

The  most  significant  difference  between  the  formulations  of  the  CCR  model  and 
BCC  model  lie  in  how  the  model  accounts  for  output  /  input  returns  to  scale.  The  CCR 
model  assumes  constant  returns  to  scale,  such  that  an  increase  in  an  input  parameter  has 
a  corresponding  and  constant  increase  (or  benefit)  in  an  output  parameter.  The  BCC 
model  accounts  for  variable  returns  to  scale.  This  occurs  when  the  benefit  in  output  per 
unit  input  decreases  as  the  input  becomes  larger.  The  basic  BCC  formulation,  given  in 
Figure  2,  is  quite  similar  to  the  CCR  model  except  for  the  addition  of  a  constant,  c,  in  the 
objective  function. 
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Indices: 

i 


Output  Types  (1,2,...I) 
Input  Types  (1,2,..  .J) 
DMUs  (1,2, ...K) 


j 

k 


Data: 

%kj 

Value  of  input  j  for  DMU  k 

yia 

Value  of  output  i  for  DMU  k 

s 

A  small  constant. 

Variables: 

vj 

Weight  on  input  j  (unitless) 

Wj 

Weight  on  output  i  (unitless) 

Constants: 

ck 

Constant  for  each  DMU  k 

Formulation: 

For  each  DMU  o,  (o  =  1,2,...K) 

^  yom+Co 


Maximize 


XojVj 


Subject  to  constraint: 

^  yktWi+Co 


<1,  fork  =  l,2,...K 


Wj>  € ,  fori  =  1,2,...I 
Vj  >  £ ,  for  j  =  1,2,...  J 


Figure  2.  BCC  DEA  Model  Formulation  (Norman,  1991,  p.  247) 


C.  CONCLUSION 

Data  Envelopment  Analysis  provides  an  appropriate  approach  for  determining  the 
efficiency  of  Contract  Administration  Offices.  The  first  and  most  challenging  step  in 
formulating  a  DEA  model  is  the  selection  of  the  input  and  output  measures.  The 
following  chapter  will  determine  the  best  collection  of  measures  from  a  field  of  available 
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statistics  to  use  in  the  two  CAO  models.  The  measures  for  each  model  will  be  selected 


separately,  so  that  only  the  most  appropriate  data  is  selected  for  those  models.  Finally, 
the  relationship  between  output  and  input  measures  is  investigated  in  order  to  select  the 
proper  DEA  formulation:  CCR  (constant  returns  to  scale)  or  BCC  (variable  returns  to 
scale).  Once  the  model  is  formulated,  insights  into  the  efficiency  of  DCMC  are  made  and 
improvement  strategies  observed.  These  efficiency  scores  can  be  used  by  management  to 
assist  in  determining  the  proper  resource  requirements  for  these  offices. 
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III.  MODEL  FORMULATION 


Selection  of  inputs  and  outputs  for  Data  Envelopment  Analysis  models  is  at  the 
center  of  the  model  formulation.  The  challenge  resides  in  determining  the  proper 
measures  that  most  accurately,  completely,  yet  succinctly  describe  the  true  effectiveness 
of  Contract  Administration  Offices  in  the  two  models  to  be  formulated  (Plant  Model  and 
Area  Model).  In  this  chapter,  after  briefly  describing  the  data,  its  origin,  and  some  if  its 
characteristics  and  limitations,  the  relationship  of  the  inputs  to  the  outputs  will  be 
analyzed  in  order  to  determine  their  possible  contribution  in  the  two  DEA  models. 

Public  sector  organizations  often  are  forced  to  rely  on  subjective,  rather  than 
objective  measures  of  performance  because  what  may  be  assumed  to  be  outputs,  are  often 
assigned  requirements,  not  under  the  control  of  the  organization.  That  is,  increases  or 
reductions  in  inputs  may  not  affect  outputs  at  all.  The  effect  of  reducing  an  input,  while 
increasing  efficiency  score,  may  only  reduce  the  quality  of  service  of  that  particular 
office.  In  the  case  of  DCMC,  a  combination  of  subjective  and  quantitative  measures  are 
available  for  analysis  and  possible  inclusion  in  the  DEA  model. 

Chames,  in  his  seminal  work  with  DEA,  stated  that  in  general,  not  more  than  one 
measure  should  be  included  for  every  three  evaluated  Decision  Making  Units  (DMU) 
(Chames,  1994,  p.  73).  (As  the  number  of  measures  approaches  the  number  of  DMUs, 
the  proportion  of  DMUs  evaluated  as  efficient  will  approach  100%.  Each  added 
measure,  in  effect,  provides  another  factor  in  which  the  DMU  attempts  to  maximize  its 
efficiency  score).  As  the  DCMC  models  contain  31  DMUs  for  the  Area  Model  34 
DMUs  for  the  Plant  Model,  every  effort  to  maintain  fewer  than  10  measures  of  inputs 
and  outputs  in  total  must  be  taken. 
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Finally,  a  positive  relationship  should  exist  between  selected  inputs  and  outputs. 
As  the  inputs  in  the  data  set  generally  increase,  the  expected  result  is  an  increase  in  the 
associated  outputs. 

A.  DATA  DESCRIPTION  AND  INITIAL  MEASURE  SELECTION 

A  panel  of  subject  matter  experts  at  DCMC  chose  candidate  input  and  output 
measures  for  further  analysis  based  on  their  knowledge  of  the  core  performance 
requirements  of  these  offices.  They  determined  that  all  CAOs  perform  certain  core  tasks 
that  account  for  the  vast  majority  of  the  workload  and  additionally  are  evaluated  on  key 
quality  measures.  Thus,  the  candidate  output  factors  measure  aspects  of  workload  and 
performance  quality  (Russell,  1998). 

1.  Databases 

Three  databases  maintained  by  DLA  provide  the  candidate  input  and  outputs:  The 
Performance  Labor  Accounting  System  (PLAS),  Mechanization  of  Contract 
Administration  Services  (MOCAS)  database,  and  the  Automated  Metric  System  (AMS) 
database.  All  data  analyzed  was  collected  in  fiscal  year  1998  and  usually  includes  totals 
for  the  year.  In  certain  cases  (as  noted)  averages  or  other  measures  are  used.  In  general, 
non  reported  data  is  given  a  default  value  of  zero. 

The  Performance  Labor  Accounting  System  (PLAS)  records  every  hour  worked 
by  an  individual  staff  member  for  a  particular  CAO  and  records  the  type  of  work  he  or 
she  is  accomplishing  for  that  hour.  Employees,  regardless  of  position,  are  required  to 
record  hours  worked  according  to  the  kind  of  contract  (contract  kinds  are  defined  as 
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Systems  Acquisition,  Research  and  Development,  Maintenance  and  Facilities,  Service, 
Supply,  and  Subcontracts),  and  by  the  type  of  work  (inspections,  audits,  cost  analysis, 
etc.).  The  PLAS  data  base  enables  rapid  computation  of  the  number  of  hours  worked  at  a 
particular  office  in  a  given  time  period  (DCMC  Unit  Cost,  1998,  p.  6). 

The  MOCAS  database  tracks  contracts  by  kind  and  their  value.  This  data  base 
provides  a  count  of  the  contracts  managed  by  each  CAO  as  well  as  their  outstanding 
dollar  value  (unliquidated  obligations).  These  two  measures  serve  as  candidates  for 
workload  (number  of  contracts  and  value  of  contracts)  and  may  be  used  as  outputs  in  the 
model,  (Defense  Logistics  Agency,  1998). 

The  Automated  Metric  System  (AMS)  provides  key  measures  of  performance  to 
DCMC  management  on  each  CAO.  Certain  key  measures  that  Defense  Contract 
Management  Command  uses  for  management  decisions  are  the  following:  Overhead 
costs  per  hour  of  mission  related  work,  customer  satisfaction  rate,  contractor  on-time  rate, 
and  the  value  of  savings  to  the  government  resulting  for  actions  taken  by  a  particular 
CAO.  (DLA,  1998).  Each  of  these  measures  are  candidate  outputs  for  the  efficiency 
model. 


2.  Candidate  Inputs  and  Outputs 

The  DCMC  provided  ten  separate  data  elements  for  each  CAO.  These  measures 
will  each  be  explained  in  this  section  and  are  summarized  as  follows:  Hours  Worked, 
Performance  Based  Activity  Measure,  Customer  Satisfaction  factor,  On-Time  rating. 
Supply  Contracts  Managed  Per  Month,  Other  Contracts  Managed  Per  Months,  System 
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Acquisition  Contracts  Value  Managed  Per  Months,  and  Return  on  Investment,  and 
Overhead  Costs  per  hour. 

a.  Inputs 

The  total  number  of  hours  worked  in  a  year  can  be  further  divided  into 
component  parts  by  contract  kind.  Because  DEA  methodology  considers  each  input  and 
output  equally,  the  DEA  model  may  base  an  efficiency  score  on  a  single  input  measure, 
such  as  total  hours  for  one  contract  kind.  For  this  reason,  the  subject  matter  experts 
specified  that  the  input  measure  be  total  hours  worked  in  a  given  year.  The  most 
complete  and  current  data  is  for  fiscal  year  1998  (DCMC  Unit  Cost,  1998,  pp.  4, 10-12). 

b.  Outputs 

Considerable  effort  has  been  spent  at  DCMC  in  an  attempt  to  properly 
quantify  the  products  of  CAO  efforts.  A  difficulty  in  quantifying  products  is  the  fact  that 
CAOs  do  widely  varying  work  based  on  differing  contract  kinds  and  contractors 
overseen.  The  most  common  output  measure  employed  by  DCMC  is  simply  a  count  of 
the  number  of  contracts  managed  in  a  given  month  (contracts  managed  per  month  or 
CMM).  This  measure,  provided  by  the  MOCAS  data  base,  is  derived  from  the  number  of 
contracts  that  have  seen  activity  at  the  close  of  a  particular  month.  Using  CMM  as  a 
production  unit  has  obvious  drawbacks  because  it  does  not  take  into  account  the  size,  risk 
or  complexity  of  a  particular  contract.  For  example,  contracts  for  major  new  systems 
(tanks,  trucks,  ships,  etc.)  normally  have  a  great  deal  of  complexity,  risk,  and  associated 
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work  where  recurrent  supply  or  facilities  contracts  do  not  (DCMC  Unit  Cost,  1998,  pp. 
10-11). 

As  a  means  to  partition  the  data  set,  contracts  are  categorized  as  one  of 
three  kinds.  System  acquisition.  Supply,  and  Other  contracts.  Systems  acquisition 
contracts,  the  most  complex  and  high  dollar  valued  contracts,  are  not  normally  measured 
in  terms  of  CMM.  Because  of  their  relatively  low  quantity  and  high  value,  they  are 
considered  as  a  separate  category.  Supply  contracts  are  the  greatest  single  subset  by 
contract  kind  in  the  CMM  data  set,  making  up  28%  of  the  total  count  of  contracts  within 
DCMC.  To  provide  further  insight  into  components  of  the  eventual  efficiency  score, 
supply  contracts  were  also  separated  from  the  total  and  used  as  a  candidate  output. 

The  remaining  contract  kind,  systems  acquisition  is  measured  by  its  size  or 
value  remaining  to  be  filled  for  the  government.  This  unfilled  obligation  is  called 
“unliquidiated  obligations”  in  DCMC  literature  and  is  measured  in  units  of  millions  of 
dollars. 

The  Return  on  Investment  (ROI)  is  the  sum  of  all  savings  to  the 
government  as  a  result  of  actions  taken  by  a  particular  CAO.  Examples  of  some  of  these 
actions  are  price  negotiations,  production  process  improvements,  refunds,  costs 
recaptured  through  litigation,  and  compensation  for  the  unauthorized  use  of  government 
property.  This  output  is  a  direct  result  of  cost  saving  measures  taken  by  CAOs  and  is 
measured  in  millions  of  dollars  in  savings  (DCMC  Unit  Cost,  1998,  p.  24). 

The  Customer  Satisfaction  rating,  provided  by  the  AMS  database,  is  a 
subjective  measure  of  performance  graded  by  purchasing  authorities,  such  as  government 
program  managers,  whenever  they  request  and  utilize  CAO  services.  Each  office 
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received  a  score  between  one  and  six,  where  six  was  a  perfect  score  (DCMC  Metric 
Guide,  1998,  p.  81).  Unlike  most  other  data  elements,  this  measure  is  an  average  score 
that  was  tallied  when  more  than  one  rating  was  received  in  FY98.  Unfortunately,  the 
reporting  rate  is  estimated  to  be  only  20%  (Kuhl,  1998)  and  fully  12%  of  all  offices  have 
no  rating.  Further,  little  relation  between  the  candidate  input  (total  hours  worked)  and 
customer  satisfaction  exist  (as  will  be  shown).  However,  this  is  the  only  measure  of 
CAO  service  quality  available;  certainly  an  important  aspect  of  efficiency.  Despite  its 
shortcomings,  management  at  DCMC  feel  that  the  customer  satisfaction  rating  is  a 
significant  factor  to  include  in  overall  performance  evaluations. 

The  On-time  rate  factor  for  a  CAO  is  a  measure  describing  each  managed 
contractor’s  ability  to  meet  production  time  lines  set  forth  in  their  contract  with  the 
government.  CAOs  influence  the  on-time  rate  indirectly  by  providing  timely  and 
appropriate  services  to  facilitate  the  rapid  fulfillment  of  the  contract.  This  is  also  an 
average  rate  based  on  contract  deliveries  for  FY98  (DCMC,  Metrics  Guide,  1998,  p.83). 

3.  Environmental  factors 

DCMC  selected  two  other  data  elements  as  candidates  for  variables  in  the  model. 
However,  neither  of  these  are  truly  an  input  or  an  output  even  though  they  may  have  a 
descriptive  nature  regarding  the  efficiency  of  CAOs.  As  environmental  factors,  these 
measures  are  useful  in  providing  explanations  about  the  behavior  of  the  model  and  in  one 
case,  may  be  included  in  the  model,  although  not  as  a  unique  measure. 

The  factor  called  the  Performance  Based  Activity  Measure  (PBAM)  is  a  measure 
of  the  relative  risk  an  individual  office  must  work  under  (DCMC,  1996,  p.  45).  To 
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measure  this  risk,  factors  such  as  experience  level  of  contractors,  complexity  of  contracts, 
and  administrative  overhead  for  particular  offices  based  on  their  assigned  contracts  were 
scored  to  show  that  higher  risk  requires  more  inputs.  Offices  were  assigned  a  rating 
between  3  and  9  with  higher  factor  equating  to  higher  relative  risk.  The  implication  is  that 
offices  with  higher  risk  factors  require  more  input  for  proper  and  adequate  management. 
In  terms  of  the  efficiency  model,  an  office  with  a  higher  PBAM  rating  can  be  expected  to 
require  more  input  resources  to  gain  the  same  quantifiable  outputs  as  offices  with  lower 
PBAM  rating. 

To  accommodate  this  factor  in  the  model,  each  output  can  be  scaled  by  the  PBAM 
factor  (or  each  input  by  the  inverse  of  PBAM).  The  effect  of  the  scaling  is  to  normalize 
the  data  with  respect  to  risk.  Those  offices  that  “expect”  to  have  more  inputs  for  the 
same  quantifiable  output,  have  thus  been  equalized  with  their  counterparts  that  have  a 
lower  associated  risk.  Once  the  DEA  model  is  completed  and  evaluated,  the  PBAM 
scaling  factor  can  be  multiplicatively  removed  to  determine  true  improvement  strategies. 

The  second  environmental  factor  is  a  measure  of  indirect  costs  per  hour  for  each 
CAO.  This  can  be  analyzed  as  an  overhead  cost  per  hour  of  doing  business.  Examples  of 
overhead  costs  are  hours  billed  as  a  result  of  training  or  clerical  work  not  related  directly 
to  contract  administration.  This  factor,  too,  does  not  resolve  itself  as  an  input  or  an 
output  but  is  rather  a  descriptive  measure  of  management  practice.  Once  efficiency 
scores  are  determined  it  may  serve  as  a  comparative  tool  to  determine  if  a  correlation 
exists  between  efficient  offices  and  their  associated  overhead  costs  or  if  a  “best  practice” 
or  target  overhead  cost  exists. 
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For  simplicity,  each  of  the  two  candidate  inputs,  six  candidate  outputs,  and  two 
environmental  factors  are  labeled  in  a  brief  format  from  the  following  table.  Each  is 
referred  to  throughout  this  text  by  their  abbreviated  names. 


Type 

Description 

Units 

Abbreviation 

Input 

Total  hours  recorded  by  CAO  in 
fiscal  year  1998 

hours 

Hours 

Input 

Total  hours  recorded  scaled  by 
inverse  of  PB  AM  factor 

hours 

S.hours 

Output 

Customer  satisfaction  rating 

unitless 

CustSat 

Output 

On  -  time  delivery  factor 

%  on  time 

Ontime 

Output 

Millions  of  dollars  of  unliquidated 
contract  value  managed  per  month, 
for  systems  acquisition  contracts 
only.  (Total  for  fiscal  year  1998) 

dollars 

(million) 

MUMM 

Output 

Supply  contracts  managed  per  month 
(Total  for  fiscal  year  1998) 

contracts 

SupCMM 

Output 

Contracts  managed  per  month  for  all 
other  contract  kinds.  (Total  for  fiscal 
year  1998) 

contracts 

OthCMM 

Output 

Return  on  investment  (Total  fiscal 
year  1998) 

dollars 

(million) 

ROI 

Environmental 

Performance  Based  Activity  Measure 

unitless 

PBAM 

Environmental 

Overhead  costs  per  billed  labor  hour 

dollars 

Overhead 

Table  1.  Candidate  Measures 


B.  ANALYSIS  OF  CANDIDATE  MEASURES 

In  order  to  show  a  ratio  of  inputs  to  outputs  as  meaningful  in  a  relative  efficiency 
model,  increases  in  inputs  must  be  shown  to  increase  outputs  (Norman,  1991,  p.  69). 
Further,  if  the  CCR  model  is  to  be  used,  these  relationships  must  be  linear.  In  each 
analysis,  data  was  partitioned  into  two  sets:  One  for  geographical  area  offices,  and  one 
for  plant  representative  offices. 
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1.  Analysis  Of  Linearity 


Figures  3  and  4  show  all  pairs  of  plots  for  the  eight  measures  of  Table  1  for  Area 
Model  and  Plant  Model  respectively.  In  this  case,  clear  relationships  are  difficult  to 
ascertain  in  certain  instances.  However,  even  those  measures  with  the  most  scattered 
data  points,  such  as  OnTime,  do  show  an  increasing  pattern  with  respect  to  inputs.  No 
evidence  of  non-linearity  exists  in  any  of  the  measures  from  either  of  the  models.  Due  to 
some  instances  of  non  -  reports  in  the  CustSat  data  and  ROI  data,  these  plots  tend  to  look 
bi-modal.  Specifically,  the  customer  satisfaction  rating  tends  to  be  between  5  and  6,  or  a 
default  value  of  zero  if  no  report  is  given.  When  non-reporting  activities  are  removed 
from  both  CustSat  and  ROI,  a  more  linear  trend  appears  in  the  data  plots  for  each  model. 


Hours 

cc^ 

1 

% 

c  v  o 

0  %% 

o'*  o  o 

‘V  ... 

IK  * 

ofd 

c 

of 

* 

f 

1° 

£  Q 

M' 

S.Hours 

0  * 
o  | 

* 

0  o 

r: 

Sk 

O'-  o  rs 

gfc  0 

% 

jft 
«f  * 

M 

o  c- 

o  * 

CustSat 

o  * 

o  c- 

-a 

c  * 

*► 

c-  c 

eo 

jik  , 

c$«.4Ss>  c 

*0  0  0-5 

*  * 

*  .-8= 

0 

w  0  * 

o 

■ 

OrtTime 

€  i  % 

«rp  .  0 

1  c 

’  0 
0  3> 

II 

■ 

0  * 
r> 

-  O-V 
*  v'  0 

Aik- 

0  e 

0 

$ 

0 

s 

':<D 

c  00  u 

MU  MM 

a  c 

0^0*  ri 

d'lNoo*  0  0 

S  15 

s>. 

ft c 

% 

cw^°'  * 

<5 

0  rP 

% 

.  f 

c  |>  ^ * 

0 

0  <%s 

®  0  G 

&  A  *  , 
*  0 

ft*  « 

SupCMM 

*  ° 

l?  .3 

¥  * 

■ 

■ 

ip  ^ 

OthCMM 

8  I 

q, 

4  % 

£ 

I. 

ROI 

Figure  3.  Area  Model  Scatter  Plots,  Pairs  of  Candidate  Measures 
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A  significant  conclusion  is  therefore  drawn  that  each  output  can  be  modeled  as 
linear  with  respect  to  input  and  no  transformations  of  data  are  necessary.  Neither  do 
decreasing  returns  to  scale  need  to  be  accounted  for  in  the  DEA  model. 


Figure  4.  Plant  Model  Scatter  Plots,  Pairs  of  Candidate  Measures 


2.  Correlation  Analysis 

Because  of  the  linear  nature  of  the  data,  analysis  of  the  pairwise  correlation 

coefficients  between  inputs  and  outputs  gives  a  relative  strength  of  this  linear 

relationship.  Tables  2  and  3  show  the  correlation  between  each  measure  for  both  model 

sets.  Positive  correlation  between  inputs  and  outputs  indicates  a  strong  candidate  for 
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inclusion  in  a  DEA  model.  However,  strong  correlation  between  inputs  with  other  inputs 
and  outputs  with  other  outputs  may  indicate  a  redundancy  in  the  measurement  of  one  or 
more  of  these  factors.  Each  factor,  among  its  peers  of  inputs  or  outputs  should  measure  a 
unique  factor  to  the  greatest  extent  possible. 

As  suspected,  the  two  input  measures  are  highly  correlated  in  both  models, 
indicating  that  only  one,  not  both,  input  measures  should  be  chosen.  Further,  an  arbitrary 
selection  of  one  of  these  two  inputs  for  inclusion  in  a  model  would  not  have  a  great 
impact  on  the  overall  result  of  the  model.  The  real  goal  of  scaling  the  input  hours  is  not 
to  affect  the  average  outcome  but  to  impact  efficiency  scores  of  offices  individually. 


Hours 

S.hours 

CustSat 

Ontime 

MUMM 

OthCMM 

ROI 

Hours 

i 

.95 

-.13 

-.30 

.37 

.87 

.54 

.34 

S.hours 

.95 

1 

-.16 

-.30 

.48 

.81 

.50 

.26 

CustSat 

-.13 

-.16 

1 

.50 

.17 

-.06 

-.62 

.01 

Ontime 

-.30 

-.30 

.5 

1 

.17 

-.17 

-.30 

-.15 

MUMM 

.37 

.48 

.17 

.17 

1 

.29 

.03 

.29 

SupCMM 

.87 

.81 

-.06 

-.17 

.29 

1 

.43 

.21 

OthCMM 

.54 

.50 

-.62 

-.30 

.03 

.43 

1 

.21 

ROI 

.34 

.26 

.01 

-.15 

.29 

.21 

.21 

1 

Table  2.  Correlation  Coefficients  for  Measures,  Area  Model 


Hours 

S.hours 

CustSat 

Ontime 

MUMM 

OthCMM 

ROI 

Hours 

i 

.97 

.19 

,ii 

.36 

.49 

.64 

.40 

S.hours 

.97 

1 

.22 

.12 

.40 

.52 

.65 

.36 

CustSat 

.19 

.22 

1 

.20 

.11 

.16 

.28 

.05 

Ontime 

K91 

.12 

.20 

1 

.16 

.35 

.22 

.01 

MUMM 

.40 

.11 

.16 

1 

.13 

.01 

-.07 

SupCMM 

.49 

.52 

.16 

.35 

.13 

1 

.66 

.10 

OthCMM 

.64 

.65 

.28 

.22 

.01 

.66 

1 

.31 

ROI 

.40 

.36 

.05 

.01 

-.07 

.10 

.31 

1 

1 

fable  3.  Correlation  ( 

Coefficients  for  Measures,  Plant  Model 

The  contract  count  measures  (SupCMM  and  OthCMM)  are  also  relatively  highly 
positively  correlated  with  each  other.  This  indicates  that  offices  that  manage  a  large 
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number  of  supply  contracts  also  manage  a  large  number  of  other  contracts.  Although  it  is 
possible  that  an  aggregate  count  of  total  contracts  managed  may  provide  a  more 
meaningful  result  in  the  final  efficiency  analysis,  aggregation  also  serves  to  dilute  the 
effect  of  the  major  component  of  the  CMM  factors  (supply  contracts  count)  as  a  separate 
measure  of  output.  For  this  reason,  two  CMM  factors  will  remain.  The  similarities  in  the 
two  data  sets  are  substantial,  however,  key  differences  can  be  highlighted. 

a.  Defense  Contract  Management  Area  Offices 

In  the  Area  model,  a  high  positive  correlation  exists  between  each  input 
candidate  and  the  two  contract  count  measures.  A  somewhat  lower  correlation  with 
MUMM  as  well  as  the  ROI  factor  also  exist.  These  four  measures  seem  to  be  the  core 
output  measures  in  this  model  with  respect  to  Hours  and  S. hours.  A  negative  relationship 
exists  for  the  Customer  satisfaction  and  On-time  rate.  The  Ontime  correlation  of  -.3 
admits  that  most  likely  no  positive  relation  does  exist  with  the  input.  It  is  probable  that 
an  increase  in  hours  has  no  effect  on  any  contractor’s  ability  to  meet  his  contracted 
schedule. 

In  fact,  the  results  of  a  two  tailed  test  of  the  null  hypothesis  of  zero 
correlation  on  each  of  the  correlation  coefficients,  (at  a  .05  level  of  significance),  show 
that  most  likely  no  correlation  exists  between  ROI,  CustSat,  or  OnTime  with  either 
Hours  or  S.Hours.  (The  other  measures  show  strong  evidence  of  correlation.)  These 
three  uncorrelated  measures  constitute  half  of  the  potential  outputs  in  the  model. 
However,  this  result  is  not  completely  unexpected  as  the  current  staffing  levels  have  been 
assigned  with  the  regression  model  (discussed  in  Chapter  II)  utilizing  only  workload  as 
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the  modeled  variable.  The  three  uncorrelated  measures  have  never  been  "rewarded"  with 


additional  staff  resources  in  the  past.  It  is  possible  that  after  implementation  of  a  model 
for  staff  resourcing  that  does  reward  these  measures,  correlation  will  increase.  In  any 
case,  the  inclusion  of  customer  satisfaction  rating  and  on-time  factor  will  simply  serve  as 
another  factor  for  the  offices  with  a  high  scores  in  these  metrics  to  take  advantage  of  in 
the  DEA  methodology.  Additionally,  they  are  the  only  qualitative  measures  available, 
and  as  such,  must  be  included  in  the  model  formulation.  Chapter  IV  will  consider  some 
measures  to  limit  their  overall  effect  on  the  efficiency  scores  calculated. 

b.  Plant  Representative  Offices 

Relationships  in  the  Plant  Representative  model  are  much  more  clear. 
Every  input  and  output  relationship  is  indeed  positive  although  those  relationships  vary 
greatly  in  their  magnitude.  The  greatest  correlation  exists  in  what  appear  to  be  the  core 
measures  with  respect  to  hours  and  scaled  hours:  MUMM,  SupCMM,  OthCMM,  and 
ROI.  But  a  different  correlation  than  in  the  area  model  exists  with  customer  satisfaction 
rate.  This  may  be  in  part  due  to  the  lower  reporting  rate  among  plant  offices  for  this 
factor  (82%  in  plants  versus  94%  for  area  offices)  and  higher  average  score. 

The  on-time  rate  factor,  while  positively  correlated  with  scaled  hours  at 
.12  or  with  hours  at  .1 1  is  still  the  lowest,  by  far,  of  all  other  measures.  The  results  of  a 
two  tailed  test  of  zero  correlation  on  each  of  the  correlation  coefficients,  (at  a  .05  level  of 
significance),  show  that  likely  no  correlation  exists  between  the  CustSat  or  OnTime 
measures  with  either  Hours  or  S. Hours.  Unlike  the  Area  model,  ROI  does  show  evidence 
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of  correlation.  As  in  the  Area  Model,  these  factors  may  be  included  in  the  model,  with 
restrictions  on  its  overall  effect. 

c.  The  Effect  Of  Management 

As  noted  earlier,  CAOs  have  been  resourced  according  to  a  regression 
model  utilizing  workload  against  staffing.  It  is  likely  that  this  method  of  resourcing  has 
served  to  undermine  the  correlation  of  output  measures  not  related  to  that  regression 
model  (on-time  rate  and  customer  satisfaction).  Further,  even  though  measures  of 
quality  were  not  required  in  the  regression  model,  they  are  absolutely  necessary  in  an 
efficiency  model  where  outputs  are  assigned  rather  than  under  the  control  of  office 
management.  For  these  reasons,  quality  measures,  regardless  of  their  negligible 
statistical  correlation  with  the  input  must  be  included  in  both  models. 


C.  DEA  MODEL  FORMULATION 

The  linearity  of  every  output  /  input  plot  implies  that  a  constant  return  to  scale 
relationship  exists  among  them.  Evidence  suggests  that  a  CCR  DEA  model  formulation 
best  fits  the  DCMC  data  for  efficiency  evaluation.  The  general  CCR  model  general 
formulation  shown  in  Chapter  II  can  be  reformulated  as  a  linear  program  with  the  specific 
parameters  and  indices  from  the  DCMC  data  as  follows: 
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Indices: 

i  Output  Types  (SupCMM,  OthCMM,  MUMM,  ROI,  CustSat,  OnTime) 

j  Input  Types  (Hours  or  S.Hours) 

k  DMUs  (Contract  Administration  Offices  (1  to  34  for  Plant  Model  and  1  to  3 1  for 

Area  Model)) 

Data: 

Xkj 

Value  of  input  j  for  DMU  k 

ya 

Value  of  output  i  for  DMU  k 

8 

A  small  constant. 

Variables: 

vj 

Weight  on  input  j  (unitless) 

w, 

Weight  on  output  i  (unitless) 

Formulation: 

For  each  DMU  o ,  o  =  1,2,. ..K 

Maximize  zc  =  £  yCi  w* 

i 

Subject  to: 

£  (Xkj  Vj )  -  £  (yid  Wi)  <0,  k=  1,2,...K 

j  i 

Z  Xoj  Vj  =  1 

j 

Vj  >  e,  j=  1,2,. ..J 

w;  >  e,  i=  1,2,...I 

Figure  5.  Linearized  CCR  Formulation  (Norman,  1991,  p.  236) 


A  total  of  65  linear  program  solutions  are  required  for  the  two  models  for  DCMC, 
one  for  each  DMU. 


D.  CONCLUSIONS 


The  DEA  models  for  both  Plant  and  Area  offices  will  be  CCR  DEA  models  that 

contain  either  total  hours  or  scaled  hours  as  the  input.  Comparisons  between  the  two 

■ 

input  factors  will  be  considered  after  model  formulation.  Outputs  in  each  model  will  be 
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CustSat,  SupCMM,  OthCMM,  MUMM,  ROI  and  OnTime.  Even  though  correlation 
analysis  of  the  “quality”  type  measures  do  not  suggest  inclusion  in  the  model,  they  are 
absolutely  necessary  for  implementation  in  this  model  because  all  other  outputs  are 
assigned  rather  than  under  the  control  of  the  individual  office  management  team.  The 
qualitative  measures  are  required  to  ensure  that  the  assigned  outputs  are  being  performed 
well  (Ganley,  1992,  p.36). 

The  following  chapter  will  outline  the  final  model  formulation  for  each  office 
type  and  discuss  the  results  of  the  CCR  DEA  models  for  both  plant  and  area  models. 

Also  suggested  improvement  strategies  for  individual  inefficient  offices  will  be  analyzed. 
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IV.  MODEL  RESULTS  AND  ANALYSIS 


The  two  DEA  efficiency  models  (Plant  Office  Model  and  Area  Office  Model)  are 
created  with  one  input  parameter  and  six  output  parameters.  Because  of  the  general 
linear  relationship  between  the  input  and  output  values,  the  DEA  formulation  that 
assumes  constant  returns  to  scale  was  used  to  express  an  optimally  weighted  ratio  of 
outputs  to  inputs.  The  CCR  formulation,  outlined  in  Chapters  II  and  HI  produced  optimal 
efficiency  scores  for  each  evaluated  Contract  Administration  Office. 

Here,  Decision  Making  Units  (DMU)  are  Contract  Administration  Offices,  the  set 
of  inputs  is  either  Hours  or  S.Hours,  and  the  set  of  outputs  are  SupCMM,  OthCMM, 
MUMM,  ROI,  CustSat,  and  OnTime.  The  65  fractional  programs,  (34  for  Plant  Model 
and  31  for  Area  Model),  which  can  be  equivalently  expressed  as  linear  programs  (LP), 
were  solved  with  the  General  Algebraic  Modeling  System  (GAMS),  utilizing  the  OSL 
solver. 

In  this  chapter,  a  final  formulation  of  the  two  models  is  presented  after  analyzing 
two  unanswered  questions  from  the  previous  chapter:  What  is  the  proper  input  measure? 
How  can  limits  on  the  un-correlated  metrics  be  imposed?  Once  the  models  are  finally 
formulated,  analysis  of  efficiency  scores  and  insights  into  improvements  for  inefficient 
offices  can  be  made. 

A.  FINAL  MODEL  FORMULATION 

1.  Effect  Of  Risk  Factor  Scaling 

The  effect  of  scaling  hours  by  risk  factor  was  unknown  prior  to  modeling  but 
expected  to  slightly  lower  efficiency  in  low  risk  offices  and  slightly  raise  efficiency  in 
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those  with  high  risk  while  not  appreciably  changing  the  average  efficiency  score  over  all 
offices.  This  should  assure  that  offices  with  high  risk  are  not  penalized  because  their 
workload  requires  a  greater  input  expenditure  by  its  very  nature  and  that  low  risk  offices 
are  not  given  an  unfair  advantage.  By  scaling  the  input  hours,  the  intended  result  is  that 
those  offices  with  low  risk  will  have  to  "work  harder"  to  achieve  efficiency  than  those 
with  high  risk. 

The  resulting  efficiency  scores  show  decisively  that  this  is  the  case.  A 
comparison  of  the  aggregated  mean  efficiency  score  for  each  model  for  each  different 
input,  shows  that  a  very  small  change  took  place  (see  Table  4).  The  change  from  un¬ 
sealed  hours  to  scaled  hours  in  the  Area  Model  resulted  in  a  slight  decrease  in  average 
score  from  76.36%  to  75.47%  while  in  the  Plant  Model  a  slight  increase  from  72.88%  to 
74.78%  was  shown.  The  conclusion  is  that  the  scaling  factor  has  little  impact  on  the 
average  efficiency  score. 

In  examining  the  score  changes  for  each  individual  office  between  the  two 
models,  the  greatest  impact  of  scaling  can  be  seen  in  the  extreme  cases.  The  five  lowest 
risk  and  five  highest  risk  offices  show  the  most  dramatic  score  change. 
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Five  Offices  with 
Lowest  Risk 

Score 

no 

Scaling 

Score 

with 

Scaling 

change 

%  change 

Area  Office  31 

100.00 

100.00 

0.00 

0.00% 

Area  Office  23 

45.88 

34.11 

-11.77 

-25.66% 

Area  Office  19 

100.00 

100.00 

0.00 

0.00% 

Area  Office  25 

78.39 

55.14 

-23.25 

-29.66% 

Area  Office  20 

98.37 

74.47 

-23.90 

-24.29% 

Five  Offices  with 
Highest  Risk 

Score 

no 

Scaling 

Score 

with 

Scaling 

change 

%  change 

Plant  Office  17 

49.48 

67.20 

17.72 

35.81% 

Plant  Office  18 

56.73 

73.21 

16.48 

29.06% 

Plant  Office  23 

55.06 

61.23 

6.17 

11.20% 

Plant  Office  9 

79.45 

90.41 

10.96 

13.80% 

Plant  Office  21 

100.00 

100.00 

0.00 

6.00% 

Table  4.  Risk  Factor  Scaling 


Note  that  offices  in  Table  4  with  low  risk  that  were  rated  efficient  in  the  un-scaled 
model  remained  so  in  the  scaled  model.  The  strength  of  these  offices  in  outputs  is 
evident  in  their  ability  to  overcome  the  scaling  up  of  its  input  hours  with  risk  factor  and 
remain  efficient  in  comparison  to  all  other  offices.  In  the  case  of  the  high  risk  office  that 
was  efficient  in  the  un-scaled  model,  a  further  decrease  (or  scale  down)  of  its  input  hours, 
has  no  further  effect  as  it  is  already  able  to  maximize  its  efficiency  score.  Also  of  interest 
is  the  fact  that  the  5  lowest  risk  offices  are  all  area  offices  and  the  5  highest  risk  offices 
are  all  plant  offices.  The  graphs  below  (Figures  6  and  7)  show  in  detail  the  individual 
score  changes  as  impacted  by  risk  factor  scaling  of  the  model. 
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Efficiency  With  Scaled  Hours  Input  m  Efficiency  With  Scaled  Hours  Input 
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igure  6.  Area  Model  Efficiency  Scores,  Scaled  Hours  vs.  Un-scaled  hours 
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Figure  7.  Plant  Model  Efficiency  Scores,  Scaled  Hours  vs.  Un-Scaled  Hours 


The  evidence  supports  the  hypothesis  that  scaling  by  risk  factor  has  the  intended 
effect  of  equalizing  the  inputs  for  all  offices  based  on  inherent  risk.  For  this  reason,  only 
models  with  the  scaled  input  hours  will  gain  further  consideration  for  employment  and 
refinement. 

2.  Effect  Of  Variable  Constraints  On  Efficiency  Scores 

A  great  advantage  of  DEA  methodology  is  the  ability  for  each  unit  to  be  evaluated 
on  its  strengths.  If  the  unit  has  only  one  strong  parameter  in  a  field  of  six,  it  is  still 
possible  to  receive  an  efficiency  score  of  one  (100%).  In  this  case,  however,  a  few 
parameters  did  not  display  a  positive  relationship  with  the  input  parameter.  An  efficient 
rating  for  an  office  utilizing  only  one  of  those  less  strongly  correlated  factors  should  be 
limited  in  order  to  gain  the  greatest  insight  from  the  eventual  scores.  The  problem  is 
easily  visualized  when  one  assumes  the  worst  case:  An  office  with  very  few  contracts  to 
manage,  with  small  dollar  value,  but  with  a  very  good  OnTime  and  CustSat  rating  can 
certainly  be  evaluated  as  "efficient".  In  fact,  the  results  of  the  models  bear  this  out. 

In  the  Area  Model,  without  constraints  on  the  maximum  values  of  the  weights, 
five  of  31  offices  rely  on  one  of  the  weakly  correlated  parameters  for  more  than  50%  of 
their  efficiency  rating.  In  one  case,  fully  94%  of  the  output  weight  (variable  value  of  .94) 
is  on  the  customer  satisfaction  rating.  The  effect  on  all  offices  are  shown  at  Figure  8  with 
the  most  constraint  sensitive  offices  labeled. 

The  Plant  Model  felt  a  significantly  greater  impact  on  efficiency  scores  with  the 
imposition  of  variable  constraints.  Plant  Model  offices  tend  to  rely  on  fewer  parameters 
than  do  the  Area  Model  Offices.  This  was  expected  because  area  offices  manage 
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contracts  from  a  wide  variety  of  contractors  and  even  wider  range  of  contract  kinds  and 
types  resulting  in  an  efficiency  score  that  combines  many  factors.  In  comparison,  the 
plant  offices,  often  with  only  one  contractor  to  service,  tend  to  be  strong  in  only  one  or 
two  performance  measures.  On  average.  Area  Model  offices  used  3.5  output  parameters 
in  their  efficiency  score  while  Plant  Model  offices  used  only  2.6.  As  a  result,  the  impact 
of  constraints  on  two  parameters  is  magnified  in  the  Plant  Model.  Of  concern,  however, 
is  the  fact  that  three  offices  in  the  Plant  Model  relied  completely  on  the  customer 
satisfaction  rating  and  fully  12  offices  (35%  of  all  plant  offices)  used  the  customer 
satisfaction  rate  for  more  than  75%  of  their  overall  efficiency  score. 

Area  Model 

Unconstrained  vs.  Constrained 


Efficiency  With  Constraints  on  3  Parameters 


Figure  8.  Area  Model  Constraint  Impact  on  Efficiency  Scores 
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Clearly,  a  limit  on  these  factors  is  in  order.  An  upper  limit  of  50%,  33%,  or  25% 
on  the  statistically  un-correlated  parameters  is  imposed  and  has  the  effect  of  drawing 
down  the  score  of  the  offices  heavily  dependant  on  them  and  forcing  more  reliance  on  the 
more  highly  correlated  metrics. 


Plant  Model 

Unconstrained  vs.  Constrained 


0  10  20  30  40  50  60  70  ,  80  90  100 

•  50%  Limit  D33%  Limit  +25%  Limit 
Efficiency  With  Constraints  on  2  Parameters 

Figure  9.  Plant  Model  Constraint  Impact  On  Efficiency  Scores 


With  the  imposition  of  constraints,  it  is  possible  for  offices  to  either  receive  the 
same  score  as  in  the  unconstrained  model  or  a  lower  score.  In  some  cases  this  lower 
score  is  dramatic;  the  result  of  an  over  reliance  on  the  un-correlated  parameters  for 
efficiency  rating.  The  effect  of  the  25%  and  33%  constraint  appear  to  be  overly 
restrictive.  The  50%  constraint  has  the  intended  effect  of  significantly  reducing  the 
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efficiency  score  for  offices  that  rely  too  heavily  on  those  metrics.  The  more  restrictive 
constraints  begin  to  adversely  impact  offices  that  do  not  seem  overly  dependant  on  those 
parameters  for  their  efficiency  score.  The  50%  upper  limit  constraint  on  CustSat, 
OnTime,  and  ROI  will  be  imposed  in  the  Area  Model,  the  same  constraint  will  placed 
only  on  CustSat  and  OnTime  in  the  Plant  Model. 

3.  The  Final  Model 

The  two  final  pieces  of  the  model  formulation  are  now  in  place.  The  models  are 
both  constant  returns  to  scale  models  that  utilize  total  hours  scaled  by  risk  factor  as  an 
input,  all  six  outputs  measures,  with  constraints  on  the  un  -  correlated  parameters 
restricting  their  weight  to  less  than  50%  of  each  overall  efficiency  score.  Analysis  of  the 
efficiency  scores  and  an  examination  of  improvement  methods  for  inefficient  offices  can 
now  be  examined. 

B.  EFFICIENCY  SCORE  SUMMARY 

1.  Overview  Of  Results 

Efficiency  scores  for  all  offices  are  at  Appendix  A.  The  results  of  each  model  are 
summarized  in  the  following  tables  and  figures. 


Area 

Plant 

Number  Efficient 

7 

8 

Mean 

74.71% 

66.58% 

Median 

72.07% 

69.99% 

Standard  Deviation 

21.46% 

27.30% 

Range 

80.67% 

99.38% 

Minimum 

19.32% 

0.61% 

Maximum 

100% 

100% 

Table  5.  Summary  Statistics 
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Examination  of  the  summary  scores  leads  one  to  assume  that  the  model  results 
were  quite  similar.  However,  closer  inspection  of  score  frequency  diagrams  tells  a 
different  story.  While  most  of  the  Area  Model's  offices  have  scores  above  50%,  the  Plant 
Model  has  a  significant  amount  of  low  scoring  offices. 
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Figure  10.  Area  Model  Score  Frequency  Stem  and  Leaf  Display 


Stem:  Tens,  Leaf:  Ones 


0 

1 

1 

3 

4 

2 

8 

3 

4 

4 

5 

5 

5 

0 

1 

2 

3 

6 

9 

6 

0 

1 

1 

7 

0 

0 

1 

3 

6 

6 

7 

8 

3 

7 

9 

9 

10 

0 

0 

0 

0 

0 

0 

0 

0 

Figure  11.  Plant  Model  Score  Frequency  Stem  and  Leaf  Display 


Inspection  of  the  data  for  these  four  poor  performers  reveals  insight  into  their  low 

efficiency  score.  While  in  all  cases,  their  input  hours  value  was  near  the  average  of  all 

other  plant  offices,  all  four  reported  zero  for  their  MUMM  statistic.  That  is:  the  value  of 

their  contracts  managed  was  zero.  Because  MUMM  data  is  only  collected  for  one  kind  of 
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contract  (Systems  Acquisition),  this  fact  alone  is  not  terribly  unusual.  However,  closer 
inspection  of  the  input  data  shows  hours  logged  specifically  for  Systems  Acquisition 
contracts.  This  leads  to  the  assumption  that  a  Systems  Acquisition  contract  value  greater 
than  zero,  no  matter  how  small,  should  be  in  the  MOCAS  database  for  these  offices. 
Additionally,  their  contract  count  statistics,  while  not  zero,  were  very  low  compared  to 
other  offices.  The  lower  value  of  the  quantitative  output  measures  for  these  four  offices 
(MUMM,  SupCMM,  and  OthCMM)  coupled  with  the  imposed  constraint  on  the 
qualitative  measures  ensured  their  low  score. 

It  seems  clear  that  some  type  of  data  collection  error  is  in  evidence  in  either  the 
PLAS  or  MOCAS  database  for  these  few  offices.  Fortunately,  this  does  not  undermine 
the  methodology.  While  an  artificially  high  measure  in  the  data  may  perturb  the  results 
of  the  model,  incorrect  low  scores  will  not.  In  any  case,  inferences  about  the  efficiency 
of  these  four  offices  are  difficult  to  ascertain. 

2.  Quantitative  vs.  Qualitative  Measures 

Plant  and  Area  offices  generally  rely  on  different  subsets  of  outputs  in  optimizing 
their  efficiency  scores.  The  Area  Model  consistently  relied  on  quantitative  measures  to 
produce  efficiency  scores  for  its  office.  The  early  data  analysis  in  Chapter  II  certainly 
showed  that  a  significant  relationship  exists  with  these  measures  while  the  qualitative 
measures  failed  to  show  any  correlation  was  present  in  hypothesis  testing.  In  Figure  10, 
The  portion  of  quantitative  measures  making  up  the  efficiency  score  of  each  offices  is 
plotted  against  the  total  efficiency  score.  (For  example,  the  portion  of  an  efficiency  score 
of  75%  may  be  made  of  50%  quantitative  measures  and  25%  qualitative  measures.) 
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Those  offices  lying  along  the  "100%  Dependence"  line  base  their  entire  efficiency  score 
on  the  quantitative  measures.  The  farther  offices  lie  from  that  line,  the  more  they  are 
dependant  on  the  qualitative  measures.  Figure  13  shows  the  Area  Model  qualitative 
measures  vs.  their  overall  efficiency  score.  This  graph  is  the  complement  or  "mirror"  of 
Figure  12.  Those  offices  with  a  high  dependence  on  qualitative  measures  are  now  nearer 
the  100%  dependence  line. 


Area  Model 

Efficiency  vs.  Quantitative  Measures 


Figure  12.  Area  Model  Efficiency  Scores,  Quantitative  Contribution 
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Area  Model 


Weighted  Sum  of  CustSat,  ROI,  OnTime 
Figure  13.  Area  Model  Efficiency  Scores,  Qualitative  Contribution 

Inspection  of  the  same  plots  with  the  Plant  Model  shows  that  quite  a  different 
parameter  combination  is  at  work.  In  those  offices  that  rely  heavily  on  either  quantitative 
or  qualitative  measures,  more  favor  the  quality  type  measures.  However,  no  clear  trend  is 
admitted  in  the  Plant  Model.  It  seems  that  the  Plant  offices  use  a  variety  of  parameter 
combinations  to  calculate  efficiency  scores. 
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Figure  14.  Plant  Model  Efficiency  Scores,  Quantitative  Contribution 


Figure  15.  Plant  Model  Efficiency  Scores,  Qualitative  Contribution 
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C.  IMPROVEMENTS  FOR  THE  INEFFICIENT 


For  an  office  to  improve  its  efficiency  score  it  must  either  reduce  inputs,  increase 
outputs,  or  both.  By  examining  an  inefficient  office's  reference  set,  improvement  options 
can  be  identified. 


1.  Reference  Sets 

The  reference  set  is  the  set  of  efficient  units  (offices)  that  an  inefficient  unit  has 
been  most  directly  compared  with  when  calculating  its  efficiency  rating.  It  contains  the 
efficient  units  which  have  the  most  similar  input  and  output  orientation  as  the  inefficient 
unit  does.  The  reference  set  offices  should  therefore  provide  examples  of  good  operating 
practice  for  it  to  emulate. 

Area  Office  29,  an  inefficient  office,  has  four  efficient  offices  in  its  reference  set: 
Offices  5, 7, 9,  and  31.  As  detailed  in  Table  6,  each  reference  set  office,  using  Office  29's 
optimal  weights  and  not  their  own,  can  still  achieve  an  efficiency  score  of  100%. 


Office 

S.Hours 

CustSat 

OnTime  MUMM  SupCMM  OthCMM 

ROI 

Score 

5 

Value 

148225 

5.74 

52.35 

3921 

46289 

24084 

13135 

Weight*Value 

1.19 

0 

21.25 

60.88 

0 

11.90 

26.40 

100 

7 

Value 

41672 

6.00 

70.45 

256 

18819 

1334 

2 

Weight*  Value 

0.33 

0 

28.59 

3.98 

0 

0.66 

0.01 

100 

19 

Value 

143907 

5.85 

61.90 

5330 

52008 

7973 

1640 

Weight*Value 

1.15 

0 

25.12 

82.75 

0 

3.94 

3.30 

100 

31 

Value 

94051 

5.65 

64.17 

3102 

15385 

904 

226 

Weight*Value 

0.75 

0 

26.04 

48.17 

0 

0.45 

0.46 

100 

29 

Value 

124816 

5.71 

74.61 

3428 

39586 

7428 

2283 

Weight*Value 

1.00 

0 

30.28 

53.22 

0 

3.67 

4.59 

91.76 

Table  6.  Reference  Set  Comparison 


Inspection  of  the  values  in  the  reference  set  allows  Office  29  to  compare  itself 

with  offices  that  operate  under  similar  conditions.  Emulating  one  these  offices  guides 
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Office  29  in  the  direction  of  efficiency.  From  the  table.  Office  29  is  fairly  consistent  with 
offices  in  its  reference  set.  Because  of  it  already  relatively  high  score  (91.76%),  no 
obvious  poor  performance  in  this  office  is  readily  seen.  However,  it  is  possible  that  a 
modest  improvement  could  be  made  by  increasing  ROI  to  align  more  closely  with 
Office  5. 

An  interesting  result  of  having  only  one  input  measure  for  each  model  is  the  fact 
that  by  holding  the  current  output  factors  constant,  the  efficiency  score  can  be  increased 
by  a  corresponding  decrease  in  S. Hours.  Simply  put,  an  input  level  that  is  91.76%  of  the 
current  level  would  increase  the  efficiency  score  to  100%.  Put  another  way.  Area  office 
29  can  become  efficient  by  reducing  its  input  hours  by  8.24%  or  about  54  thousand 
hours. 


2.  Efficient  Frontiers 

By  collapsing  the  outputs  into  two  groups,  the  quantitative  and  qualitative,  the 
efficient  frontier  can  be  seen  in  two  dimensions.  In  Figure  16,  Office  29  appears  close  to 
the  frontier.  Area  Office  29  can  increase  efficiency  by  moving  right,  increasing  its 
qualitative  factors,  or  by  moving  up,  increasing  its  quantitative  factors.  The  most  direct 
route  to  the  efficient  frontier  is  the  path  from  the  origin  through  point  29  onto  the  frontier. 
This  direction  of  improvement  shows  the  most  rapid  route  to  efficiency.  However,  in 
DCMC's  case,  quantitative  measures  are  not  under  the  control  of  local  management. 

Most  likely,  this  office  can  only  move  right  to  the  frontier  to  achieve  efficiency.  As  a 
comparison,  the  efficient  frontier  for  the  Plant  model  is  at  Figure  17. 


45 


Weighted  Sum  of  MUMM,  SupCMM, 

OthCMM/ Weighted  Input  Weighted  Sum  of  MUMM,  SupCMM,  OthCMM/ 


Area  Model 
Efficient  Frontier 

Qualitative  vs.  Quantitative  Measures 


Figure  16.  Area  Model  Efficient  Frontier 


Plant  Model 
Efficient  Frontier 

Qualitative  vs.  Quantitative  Measures 


Figure  17.  Plant  Model  Efficient  Frontier 
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D.  EFFICIENT  UNITS  AND  OVERHEAD  COST 

In  Chapter  H,  an  environmental  factor,  overhead  cost,  was  evaluated  as  a  potential 
input  or  output.  This  measure  showed  the  dollar  value  cost  per  hour  for  each  office,  that 
was  not  strictly  related  to  the  core  tasks.  These  types  of  costs  included  training  costs  and 
travel  time  costs,  among  others.  Unfortunately,  this  measure  did  not  have  the  attributes 
to  contribute  to  the  model  as  either  an  input  or  output.  However,  overhead  cost  may 
contribute  to  this  study  by  showing  an  overhead  cost  trend  in  efficient  offices.  If  such  a 
trend  exists,  inefficient  offices,  by  means  of  comparison,  can  determine  if  their  overhead 
costs  are  too  small  or  too  large.  Too  little  overhead  cost  may  indicate  that  not  enough 
time  is  spent  in  organization-sustaining  work  such  as  training.  Too  much  may  show  that 
staff  is  distracted  from  its  core  tasks  of  administering  contracts. 

Figures  18  and  19  show  efficiency  scores  plotted  with  overhead  costs.  The 
plotted  means  (dashed  lines)  for  both  efficient  offices  and  for  all  offices  seem  to  show 
that  efficient  offices  have  a  somewhat  smaller  average  overhead  cost  associated  with 
them.  However,  in  the  Plant  Model,  a  level  .05  Wilcoxon  rank-sum  test  strongly  rejects 
the  null  hypothesis  that  the  efficient  office  mean  overhead  cost  is  the  same  as  the  mean 
for  all  offices.  This  suggests  that  in  the  Plant  Model,  efficient  offices  have  a  significantly 
lower  than  average  overhead  cost.  Such  strong  evidence  does  not  exist  in  the  Area  Model 
even  though  the  average  overhead  cost  is  higher  for  inefficient  offices.  In  general,  an 
overhead  cost  significantly  higher  than  the  average  efficient  office's  may  be  an  indicator 
of  inefficiency,  although  some  exceptions  are  present. 
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Figure  18.  Plant  Model  Overhead  Costs 
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Figure  19.  Area  Model  Overhead  Costs 
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E.  CONCLUSION 


This  chapter  commenced  with  the  final  formulation  of  DEA  models  for  DCMC. 
All  the  previous  data  analysis  and  model  investigation  from  Chapters  I  through  HI  led  to 
a  DEA  model  that  assumes  constant  returns  to  scale  but  with  added  constraints  on  some 
variables.  Constraints  were  imposed  to  limit  the  effect  of  those  output  parameters  that 
displayed  little  or  no  correlation  with  the  input  parameter. 

Scores  were  then  analyzed  to  determine  by  what  means  inefficient  offices  could 
become  efficient.  Finally,  by  observing  the  overhead  costs  associated  with  each  office,  a 
trend  was  found  to  show  inefficient  offices  yet  another  possible  avenue  to  improve 
efficiency  score.  Chapter  V  will  discuss  specific  recommendations  for  further  study,  as 
well  as  recommendations  for  DCMC  should  these  models  be  implemented  to  assist  in 
policy  decisions. 
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V.  RECOMENDATIONS 


One  goal  of  this  thesis  is  to  determine  the  efficiency  of  all  65  CAOs  in  DCMC. 
Only  15  of  these  offices  are  shown  to  be  efficient.  It  is  important  to  remember,  however, 
that  each  office  is  evaluated  in  the  very  best  possible  light  with  respect  to  the  given  model 
parameters.  If  the  assumption  that  the  six  modeled  outputs  and  one  modeled  input  make 
up  the  vast  majority  of  resources  and  products  within  DCMC,  then  the  resultant  DEA 
efficiency  score  is  the  best  possible  score  for  each  office. 

A.  MODEL  VULNERABILITY 

The  model  in  its  current  form  is  vulnerable  to  data  manipulation  by  individual 
offices.  Because  some  of  the  data  elements  originate  within  each  CAO,  those  elements 
are  not  subject  to  auditing  by  higher  level  management.  Especially  vulnerable  to  internal 
manipulation  is  the  ROI  factor  and  the  total  Hours  figure.  Each  are  reported  directly  by 
the  CAOs.  While  not  to  imply  that  wholesale  manipulation  of  the  data  by  the  individual 
offices  would  arise,  certainly  some  system  of  auditing  those  reports  would  be  prudent.  In 
fact,  the  problem  with  the  Hours  figure  is  easily  solved.  A  comparison  of  the  total  Hours 
figure  with  the  number  of  employees  in  each  office  shows  these  two  measures  to  be 
nearly  identical.  Modeling  with  the  number  of  employees  provides  nearly  identical  DEA 
efficiency  scores.  Hours  were  used  in  these  models  because  DCMC  believes  that  the 
PLAS  database  gives  a  more  accurate  view  of  total  input. 

The  other  measures  used  in  the  study  are  not  generated  at  the  CAO  level  and  are 
sufficiently  robust  as  to  need  no  further  manipulation. 
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B.  DATA  COLLECTION  CONCERNS 

The  quality  of  the  customer  satisfaction  rating  is  questionable  due  to  the  low 
report  rate  by  serviced  contractors.  In  order  to  make  CustSat  a  higher  quality  measure,  a 
concerted  effort  by  management,  which  may  ultimately  require  a  new  approach  to 
customer  satisfaction  data  collection,  is  certainly  in  order.  Surely  a  bottom  line  quality 
measure  is  absolutely  essential  in  a  model  such  as  this  and  customer  satisfaction  has 
potential  to  be  that  measure. 

DEA  literature  emphasizes  that  a  causal  relationship  between  inputs  and  outputs 
must  be  present  (Norman,  1991,  p.  178).  Because  no  relation  between  input  hours  and 
these  qualitative  measures  is  evident  present,  management  should  implement  systems  to 
reward,  or  at  least  emphasize  these  measures  as  the  key  quality  metrics  for  the  entire 
organization.  Once  these  new  policies  have  been  established,  increased  correlation 
between  the  input  and  these  measures  are  expected. 

C.  INTERPRETING  THE  RESULTS 

The  result  that  50  offices  of  65  in  DCMC  have  an  efficiency  score  less  than  100% 
does  not  immediately  imply  that  an  automatic  reduction  in  personnel  strength  is  in  order 
for  all  50  offices.  Certainly,  a  best  approach  is  to  examine  an  inefficient  office’s 
reference  set,  determine  obvious  improvement  opportunities,  and  work  to  improve 
efficiency.  Ultimately,  however,  some  offices  may  indeed  require  staffing  reduction. 

Further,  this  study  did  not  take  into  account  any  type  of  "baseline"  personnel 

resource  requirement.  If  we  assume  that  every  office  needs  a  minimum  number  of 

employees  to  accomplish  daily  routine  missions,  regardless  of  the  workload  assigned,  it  is 

a  fair  argument  that  the  DEA  model  should  only  model  those  hours  above  the  baseline 
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figure.  This  baseline  figure  may  account  for  the  poor  showing  among  a  few  small  Plant 
Offices. 

D.  CONCLUSION 

The  DEA  models  presented  in  this  study  provide  a  valuable  management  tool  to 
assist  DCMC  in  determining  personnel  resource  requirements.  However,  these  models 
have  limitations.  Used  in  conjunction  with  the  current  system  of  resource  forecasting  (a 
regression  model),  offices  with  significant  resource  issues  and  problems  can  be  quickly 
identified.  The  great  added  value  with  the  DEA  models  is  the  new  visibility  over  each 
office's  strengths  and  weaknesses.  The  rapid  identification  of  these  weaknesses  allows 
management  to  focus  on  problems  and  solve  them  quickly  without  a  great  time 
investment  searching  for  possible  causal  factors. 
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APPENDIX.  EFFICIENCY  SCORES 


Each  Model  (Plant  Office  and  Area  Office)  was  solved  with  various  formulations. 
The  resulting  scores  below  detail  the  three  final  formulations  discussed  in  the  text. 

Model  1  uses  the  un-scaled  input  of  Hours  with  all  six  outputs.  No  constants  are 
placed  on  any  output  parameters. 

Model  2  uses  the  scaled  hours  input  (S.Hours).  The  outputs  are  identical  to 
Model  1. 

Model  3  uses  the  S.Hours  as  input  and  constrains  the  qualitative  output  measures 
(CustSat,  OnTime,  and  ROI)  to  less  than  50%  of  each  efficiency  score.  This  model  was 
selected  for  further  analysis. 
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CAO 

MODEL  1 

MODEL  2 

MODEL  3 

1 

45.51 

30.64 

19.33 

2 

71.87 

74.12 

74.12 

3 

100.00 

100.00 

100.00 

4 

60.40 

59.81 

59.81 

5 

100.00 

100.00 

100.00 

6 

62.42 

64.68 

64.68 

7 

100.00 

100.00 

100.00 

8 

100.00 

100.00 

100.00 

9 

75.39 

99.84 

99.84 

10 

87.29 

89.31 

89.31 

11 

57.59 

58.87 

57.08 

12 

69.10 

75.16 

75.16 

13 

73.67 

74.10 

74.10 

14 

70.12 

69.07 

67.38 

15 

62.74 

65.83 

65.83 

16 

53.71 

60.29 

60.29 

17 

79.18 

71.47 

71.47 

18 

92.44 

97.60 

97.60 

I  19 

100.00 

100.00 

100.00 

20 

98.37 

74.47 

68.88 

21 

62.50 

50.00 

50.00 

22 

100.00 

100.00 

100.00 

23 

45.88 

34.11 

34.11 

24 

78.57 

84.80 

84.80 

25 

78.39 

55.14 

55.14 

26 

47.44 

47.77 

47.77 

27 

70.31 

67.03 

63.97 

28 

72.33 

72.07 

72.07 

29 

80.14 

91.76 

91.76 

30 

71.68 

71.54 

71.54 

31 

100.00 

100.00 

100.00 

Table  A-l.  Area  Model  Efficiency  Scores 
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CAO 

Model  1 

Model  2 

Model  3 

1 

100.00 

100.00 

100.00 

2 

51.72 

57.49 

13.94 

3 

62.11 

61.39 

59.84 

4 

66.42 

64.55 

52.68 

5 

100.00 

100.00 

100.00 

6 

45.29 

45.15 

44.11 

7 

66.78 

70.34 

70.34 

8 

74.41 

73.49 

69.66 

9 

79.45 

90.41 

86.60 

10 

56.33 

56.32 

56.32 

11 

45.85 

52.46 

52.07 

12 

39.82 

45.11 

45.11 

13 

86.96 

96.25 

88.63 

14 

92.36 

82.70 

75.64 

15 

82.63 

71.15 

71.15 

16 

76.10 

77.24 

77.24 

17 

49.48 

67.20 

50.39 

18 

56.73 

73.21 

73.21 

19 

100.00 

100.00 

100.00 

20 

100.00 

100.00 

100.00 

21 

100.00 

100.00 

100.00 

22 

56.07 

58.85 

58.85 

23 

55.06 

61.23 

60.60 

24 

68.93 

68.35 

45.25 

25 

95.25 

91.88 

61.39 

26 

0.53 

0.62 

0.62 

27 

81.53 

96.13 

51.24 

28 

100.00 

100.00 

100.00 

29 

100.00 

100.00 

100.00 

30 

81.84 

92.40 

75.54 

31 

100.00 

100.00 

100.00 

32 

65.78 

57.98  ; 

27.61 

33 

85.88 

84.02 

83.32 

34 

54.53 

46.53 

12.72 

Table  A-2.  Plant  Model  Efficiency  Scores 
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